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APPARATUS AND METHOD FOR THREE-DIMENSIONAL 
RELATIVE MOVEMENT SENSING 

Reference to Related Patents 

5 This Application is related to the subject matter described in the 

following U.S. patents: U.S. Patent No. 5,578,813, filed March 2, 1995, issued 
November 26, 1996, and entitled FREEHAND IMAGE SCANNING DEVICE 
WHICH COMPENSATES FOR NON-LINEAR MOVEMENT; U.S. Patent No. 
5,644,139, filed August 14, 1996, issued July 1, 1997, and entitled 

1 0 NAVIGATION TECHNIQUE FOR DETECTING MOVEMENT OF 

NAVIGATION SENSORS RELATIVE TO AN OBJECT; and U.S. Patent No. 
5,786,804, filed October 6, 1995, issued July 28, 1998, and entitled METHOD 
AND SYSTEM FOR TRACKING ATTITUDE. These three patents describe 
techniques of tracking position movement. Those techniques are a component in 

1 5 a preferred embodiment described below. Accordingly, U.S. Patent Nos. 
5,578,813, 5,644,139, and 5,786,804 are hereby incorporated herein by 
reference. 

This application is also related to the subject matter described in U.S. 
Patent No. 6,057,540, filed April 30, 1998, issued May 2, 2000, and enfitled 

20 MOUSELESS OPTICAL AND POSITION TRANSLATION TYPE SCREEN 
POINTER CONTROL FOR A COMPUTER SYSTEM; U.S. Patent No. 
6,151,015, filed April 27, 1998, issued November 21, 2000, and entitled PEN 
LIKE COMPUTER POINTING DEVICE; and U.S. patent application Ser. No. 
09/052,046, filed March 30, 1998, entitled SEEING EYE MOUSE FOR A 

25 COMPUTER SYSTEM. These two related patents and patent application 
describe screen pointing devices, with embodiments that use techniques 
described in U.S. Pat. Nos. 5,578,813, 5,644,139, and 5,786,804. U.S. Patent 
Nos. 6,057,540 and 6,151,015, and U.S. patent application Ser. No. 09/052,046, 
filed March 30, 1998, entitied SEEING EYE MOUSE FOR A COMPUTER 

30 SYSTEM, are hereby incorporated herein by reference. 
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The Field of the Invention 

This invention relates generally to optical sensor devices. This invention 
relates more particularly to an optical sensor device for sensing relative 
movement in three dimensions. 

Background of the Invention 

The use of a hand operated pointing device for use with a computer and 
its display has become almost universal. By far the most popular of the various 
devices is the conventional (mechanical) mouse, used in conjunction with a 
cooperating mouse pad. Centrally located within the bottom surface of the 
mouse is a hole through which a portion of the underside of a rubber-surfaced 
steel ball extends. Interior to the mouse are rollers, or wheels, that contact the 
ball at its equator and convert its rotation into electrical signals representing 
orthogonal components of mouse motion. These electrical signals are coupled to 
a computer, where software responds to the signals to change by a AX and a AY 
the displayed position of a pointer (cursor) in accordance with movement of the 
mouse. 

In addition to mechanical types of pointing devices, such as a 
conventional mouse, optical pointing devices have also been developed, such as 
those described in the incorporated patents and patent application. In one form 
of an optical pointing device, rather than using a moving mechanical element 
like a ball in a conventional mouse, absolute movement between an imaging 
surface, such as a finger or a desktop, and photo detectors within the optical 
pointing device, is optically sensed and converted into movement information. 

The photo detectors in optical pointing devices are typically implemented 
in a flat, two-dimensional array. The array of photo detectors is capable of 
measuring absolute two-dimensional movement. As the array moves across an 
image, or the image moves across a stationary array, motion can be detected by 
comparing successive images. The sensed motion is in terms of the number of 
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pixels that the image on the pixel array has moved. The array is at a fixed 
distance and a fixed angle from the surface being imaged, so the motion that is 
sensed is absolute (within the error tolerance of the system). The surface or 
scene that is imaged by an optical pointing device is a near-field image, meaning 
5 that the surface is very close to the sensor array. 

If optics are used that allow the sensor array to focus on far-field objects, 
then the distance to the objects is not known and the sensed movement is 
relative. It would be desirable to provide a three-dimensional relative movement 
O sensing apparatus using multiple conventional two-dimensional photo detector 

Q 10 arrays and appropriate optics for focusing far-field objects. 

m 

M Summary of the Invention 

One form of the present invention provides an apparatus for sensing 
three-dimensional relative movement. The apparatus includes a movable motion 
nj 1 5 sensor including a first and a second two-dimensional array of photo detectors. 

'%! At least one lens directs far-field images onto the first and the second arrays of 

M photo detectors. The sensor is configured to generate digital representations of 

the far-field images and to generate three-dimensional relative movement data 
based on the digital representations of the far-field images. The movement data 
20 is indicative of motion of the sensor in three dimensions. 

Brief Description of the Drawings 

Figure 1 is a pictographic side view illustrating the main components of a 
prior art, optical, motion translation type screen pointer device. 

Figure 2 is a diagram illustrating a single two-dimensional photo detector 
25 array configured to capture images of far-field objects. 

Figure 3 is a diagram illustrating a photo detector array arrangement 
according to one embodiment of the present invention. 

Figure 4 is a diagram illustrating a photo detector array arrangement 
according to a second embodiment of the present invention. 
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Figure 5 is an electrical block diagram illustrating a sensor for sensing 
relative three-dimensional movement according to one embodiment of the 
present invention. 

Description of the Preferred Embodiments 

In the following detailed description of the preferred embodiments, 
reference is made to the accompanying drawings, which form a part hereof, and 
in which is shown by way of illustration specific embodiments in which the 
invention may be practiced. It is to be understood that other embodiments may 
be utilized and structural or logical changes may be made without departing 
from the scope of the present invention. The following detailed description, 
therefore, is not to be taken in a limiting sense, and the scope of the present 
invention is defined by the appended claims. 

Figure 1 shows a simplified representation of a side view of a prior art 
motion detection device 1 suitable for tracking the movement of a human finger 

7 pressed against a surface 5 of a transparent stud 3. A motion detection device 
Hke that shown in Figure 1 is described in detail in the above-incorporated U.S. 
Patent No. 6,057,540 (the '540 patent). The operation of motion detection 
device 1 is also summarized below. 

LED 2 emits light that is projected by lens 4 onto a region 5 that is part 
of a work surface 6 to be imaged for navigation. When the tip 6 of finger 7 is 
pressed against surface 5, the ridges of skin and any other micro texture features 
are visible in the plane of surface 5, just as if they were a part of surface 5. Lens 

8 focuses light from those features onto an array of photo detectors, which is part 
of movement sensor 9. In one embodiment, motion sensor 9 is an integrated 
circuit (IC) having an array of photo detectors, memory, and arithmetic circuits 
arranged to implement image correlation and tracking functions. An image of 
the illuminated region 6 is projected through an optical window (which may be 
transparent stud 3 itself) to a package (not shown) of integrated circuit 9 and 
onto the array of photo detectors. Lens 8 aids in the projection of the image onto 
the photo detectors. Movement sensor 9 automatically acquires and tracks any 
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suitable image. When tracking an image, movement sensor 9 produces 
incremental (X, Y) signals. 

Figure 2 is a diagram illustrating a single two-dimensional photo detector 
array 200 configured to capture images of far-field objects. Photo detector array 
200 is perpendicular to the paper in Figure 2. Photo detector array 200 is 
pointed at a scene or set of objects 204, which is an unknown distance, D, from 
array 200. A lens (not shown) focuses the portion of scene 204 within the field 
of vision 206 of the lens onto the surface of photo detector array 200. Array 200 
is a conventional photo detector array, such as that used in optical pointing 
devices, but generates images of far-field objects rather than near-field objects as 
in optical pointing devices. 

A three-dimensional (X, Y, Z) coordinate system is also shown in Figure 
2. Photo detector array 200 is aligned with the plane formed by the X-axis (into 
the paper) and the Y-axis. If photo detector array 200 is rotated about the Y-axis 
(i.e., looking at scene 204, array 200 is rotated left or right), the image on the 
surface of photo detector array 200 moves as array 200 is rotated. And the 
motion can be detected by correlating successive images. The absolute amount 
of angular motion will be unknown for several reasons, including: (1) The 
distance, D, fi-om the array 200 to the scene 204 is unknown; (2) the distance 
fi*om the array 200 to the scene 204 changes as the angle of the array 200 with 
respect to the scene 204 changes; and (3) the sizes of the features in the scene 
204 are unknown. 

Despite these unknown values, the relative angular motion of the array 
200 with respect to the scene 204 being viewed can be detected. If array 200 is 
rotated about the X-axis (i.e., looking at scene 204, array 200 is rotated up or 
down), then relative angular motion in a second dimension can be detected. 
Array 200 can also sense relative linear motion in two dimensions. 

Figure 3 is a diagram illustrating a photo detector array arrangement 300 
according to one embodiment of the present invention. Photo detector array 
arrangement 300 includes photo detector arrays 302A and 302B (collectively 
referred to as photo detector arrays 302), which are constructed at an angle, 0, 



o o 



yi 



PATENT 
PDNO: 10010478 



with respect to each other. In one form of the invention, 0 equals 90 degrees, 
although other configurations may be used. Photo detector array 3 02 A includes 
an active surface 304A having a plurality of photo detectors, and photo detector 

array 302B includes an active surface 304B having a plurality of photo detectors. 
5 Active surfaces 304A and 304B are collectively referred to as active surfaces 
304. 

In one embodiment, photo detector array arrangement 300 is rotatable in 
three dimensions, and/or is linearly movable in three dimensions. Using 
appropriate lenses 406A and 406B (shown in Figure 4) to focus far-field objects 

10 204 onto the active surfaces 304A and 304B of photo detector arrays 302 A and 
302B, relative angular motion in three dimensions (i.e., rotation of photo 
detector arrangement 300 about each of the three axes - X, Y, and Z) can be 
determined by correlating successively captured images. Arrangement 300 can 
also detect three-dimensional relative linear motion. The detection of such 

1 5 relative angular and linear motion is described in further detail below. 

Figure 4 is a diagram illustrating a photo detector array arrangement 400 
according to a second embodiment of the present invention. Photo detector array 
arrangement 400 includes photo detector arrays 3 02 A and 302B (collectively 
referred to as photo detector arrays 302), which are constructed substantially 

20 perpendicular to each other, and includes lenses 406A and 406B (collectively 
referred to as lenses 406). Photo detector array arrangement 400 is similar to 
arrangement 300, but the end of one array 3 02 A is positioned near the end of the 
second array 302B in arrangement 400, whereas in arrangement 300, the end of 
one array 302A is positioned near the middle of the second array 302B, rather 

25 than near the end of the second array 302B. 

Like arrangement 300, one embodiment of photo detector array 
arrangement 400 is rotatable in three dimensions, and/or is linearly movable in 
three dimensions. Using lenses 406 to focus far-field objects 204 onto the active 
surfaces 304A and 304B of photo detector arrays 302A and 302B, relative 

30 angular motion in three dimensions can be determined. Arrangement 400 can 
also detect three-dimensional relative linear motion. 
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In one form of the invention, lenses 406 are standard lenses for focusing 
far-field images, such as those used in many currently available inexpensive 
cameras. Such lenses typically have a depth of field of about 2 feet to infinity, 
so objects within this distance range, and within the field of view 402 or 404 of 
5 lenses 406 will be in focus. In an alternative embodiment, an auto-focus system 
is used to focus images onto arrays 302. 

One preferred motion detection technique according to the present 
invention, using photo detector array arrangement 300 or 400, involves optically 
detecting motion by directly imaging as an array of pixels the various particular 

10 optical features projected onto arrays 302. Light reflected from a set of far-field 
objects 204 is focused by lenses 406 onto the active surfaces 304 of a suitable 
array of photo detectors 302. The responses of the individual photo detectors are 
digitized to a suitable resolution and stored as a frame into corresponding 
locations within an array of memory. The digitized images are correlated to 

1 5 determine movement information. 

Figure 5 shows an electrical block diagram illustrating major 
components of a three-dimensional relative motion sensor 500 according to one 
embodiment of the present invention. Motion sensor 500 includes lenses 406, 
light-sensitive photo-sensors 548A-548C (collectively referred to as photo- 

20 sensors 548), and controller 502. Controller 502 includes electronic shutter 550 
having a first plurality of switches 551 A-551C (collectively referred to as 
switches 551) and a second plurality of switches 553A-553C (collectively 
referred to as switches 553). Controller 502 also includes a plurality of sense 
capacitors 554A-554C (collectively referred to as sense capacitors 554), 

25 multiplexer 556, amplifier 557, analog to digital (A/D) converter 558, correlator 
560, system controller 562, and shutter controller 564. 

The operation of controller 502 is primarily controlled by system 
controller 562, which is coupled to multiplexer 556, A/D converter 558, 
correlator 560, and shutter controller 564. In operation, according to one 

30 embodiment, light is directed onto a surface or set of far-field objects 204. 
Reflected light from surface 204 is directed by lenses 406 to light sensitive 
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photo-sensors 548. Photo-sensors 548 represent photo detectors in arrays 302, 
and are also referred to as photo detectors 548. In one embodiment, photo 
detectors 548 each provide a current that varies in magnitude based upon the 
intensity of Hght incident on the photo detectors 548, 

Shutter switches 551 and 553 are controlled by a shutter signal from 
shutter controller 564. Electronic shutter 550 is "open" when switches 551 are 
open and switches 553 are closed, and electronic shutter 550 is "closed" when 
switches 553 are open. Switches 551 are coupled to a reference voltage (Vref). 
When shutter switches 551 are open and shutter switches 553 are closed (i.e., 
electronic shutter 550 is open), charge accumulates on sense capacitors 554, 
creating a voltage that is related to the intensity of light incident on photo 
detectors 548. When shutter switches 553 are opened (i.e., electronic shutter 550 
is closed), no further charge accumulates or is lost from sense capacitors 554. 
Multiplexer 556 connects each sense capacitor 554 in tum to amplifier 557 and 
A/D converter 558, to amplify and convert the voltage from each sense capacitor 
554 to a digital value. Sense capacitors 554 are then discharged by closing 
switches 551 and 553. After discharging sense capacitors 554, switches 551 are 
opened so that the charging process can be repeated. 

Based on the level of voltage from sense capacitors 554, A/D converter 
558 generates a digital value of a suitable resolution (e.g., one to eight bits) 
indicative of the level of voltage. The digital values represent digital images or 
digital representations of the optical images directed by lenses 406 onto arrays 
302. The digital values may be processed, then stored as frames into 
corresponding locations within an array of memory within correlator 560. In one 
embodiment, each pixel in a frame corresponds to one of the photo detectors 
548. 

The overall number of photo detectors 548 for each array 302 is 
preferably large enough to receive an image having several features. In this way, 
images of such features produce translated pattems of pixel information as photo 
detector array arrangement 300 or 400 moves. The number of photo detectors 
548 in each array 302 and the frame rate at which their contents are digitized and 

8 
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captured cooperate to influence how fast photo detector array arrangement 300 
or 400 can be moved and still be tracked. Tracking is accomplished by 
correlator 560 by comparing newly captured sample frames with previously 
captured reference frames to ascertain movement information. 

In one embodiment, the entire content of one of the fi-ames is shifted by 
correlator 560 by a distance of one pixel successively in each of the eight 
directions allowed by a one pixel offset trial shift (one over, one over and one 
down, one down, one up, one up and one over, one over in the other direction, 
etc.). That adds up to eight trials. Also, since there might not have been any 
motion, a ninth trial "null shift" is also used. After each trial shift, those portions 
of the fi-ames that overlap each other are subtracted by correlator 560 on a pixel 
by pixel basis, and the resulting differences are preferably squared and then 
summed to form a measure of similarity (correlation) within that region of 
overlap. Larger trial shifts are possible, of course (e.g., two over and one down), 
but at some point the attendant complexity ruins the advantage, and it is 
preferable to simply have a sufficiently high frame rate with small trial shifts. 
The trial shift with the least difference (greatest correlation) can be taken as an 
indication of the motion between the two frames. 

In one embodiment, two consecutive frames from one photo detector 
array 302A are correlated, two consecutive frames from a second photo detector 
array 302B are correlated, and then the results of these two correlations are 
correlated to determine relative three-dimensional movement information. By 
correlating images firom each array 302 separately, and then correlating the 
results, photo detector array arrangement 300 or 400 can distinguish between 
rotation and translation. For example, if arrangement 300 or 400 is rotated left 
or right (e.g., rotated about the Y-axis in Figure 3), the movement information 
obtained from array 302 A will be similar to the movement information obtained 
fi*om array 302B. In contrast, if arrangement 300 or 400 is linearly translated, 
along the X-axis in Figure 3 for example, the images from array 302A will likely 
change substantially, whereas the images from array 304B may not change 
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much. So for this linear translation, the movement information obtained from 
arrays 302 A and 302B will likely be very different. 

In addition to providing digital images to correlator 560, A/D converter 
558 also outputs digital image data to shutter controller 564. Shutter controller 
5 564 helps to ensure that successive images have a similar exposure, and helps to 
prevent the digital values from becoming saturated to one value. Controller 564 
checks the values of digital image data and determines whether there are too 
many minimum values or too many maximum values. If there are too many 
minimum values, controller 564 increases the charge accumulation time of 

1 0 electronic shutter 550. If there are too many maximum values, controller 564 
decreases the charge accumulation time of electronic shutter 550. 

It will be understood by a person of ordinary skill in the art that fiinctions 
performed by motion sensor 500 may be implemented in hardware, software, 
firmware, or any combination thereof. The implementation may be via a 

1 5 microprocessor, programmable logic device, or state machine. Components of 
the present invention may reside in software on one or more computer-readable 
mediums. The term computer-readable medium as used herein is defined to 
include any kind of memory, volatile or non-volatile, such as floppy disks, hard 
disks, CD-ROMs, flash memory, read-only memory (ROM), and random access 

20 memory. 

Embodiments of the present invention may be used in virtually any 
application where detection of relative three-dimensional movement is desirable, 
including electronic gaming applications. For example, embodiments of the 
present invention may be used to control movement of an object (e.g., a screen 

25 pointer) displayed on a display screen for an electronic device. One embodiment 
of the present invention may be positioned on the top of an individual's head, so 
that when the individual's head turns to the left or right, tilts to the left or right, 
nods upward or downward, or makes other movements, corresponding three- 
dimensional relative movement data is generated and output by the sensor 500. 

30 The use of multiple two-dimensional sensor arrays provides increased 

angular resolution over that obtainable by a single array, and requires less 
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complex image processing algorithms. In addition to detection of relative three- 
dimensional movement, multiple two-dimensional sensor arrays can also be used 
to reduce errors in optical navigation systems. A large source of error in optical 
navigation systems is round-off error. Round-off errors tend to have a zero- 
5 mean, so the RMS value can be decreased by the use of multiple sensors arrays, 
such sensor array arrangements 300 or 400. 

Although specific embodiments have been illustrated and described 
herein for purposes of description of the preferred embodiment, it will be 
p appreciated by those of ordinary skill in the art that a wide variety of alternate 

^ 10 and/or equivalent implementations may be substituted for the specific 

+^ embodiments shown and described without departing from the scope of the 

ii-f ; 

present invention. Those with skill in the chemical, mechanical, electro- 
mechanical, electrical, and computer arts will readily appreciate that the present 
H invention may be implemented in a very wide variety of embodiments. This 

l==3S. 

fy 1 5 application is intended to cover any adaptations or variations of the preferred 

embodiments discussed herein. Therefore, it is manifestly intended that this 
U invention be limited only by the claims and the equivalents thereof. 
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